

Sound education policy and practice may require 
decisions based on new information collected 
on attitudes, perceptions, or facts in a school or 
community. This guide offers a five-step process to 
design effective questionnaires that follow research- 
based guidelines and can be used to survey students, 
teachers, or parents. It also lists resources on sample 
selection, questionnaire administration, and data 
collection, analysis, and presentation. 
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Summary 


To make sound decisions on policy and practice, education administrators may need to 
collect information on attitudes, perceptions, or facts in their school or community. If such 
information is not available elsewhere, educators can develop original questionnaires to 
survey students, teachers, or parents. But if the questions are not precise, the effort may 
result in inaccurate data and faulty decisions. 

This guide takes users through a five-step questionnaire development process: determin- 
ing the goal or goals of the questionnaire, defining the information needed to address 
each goal, writing the questions, reviewing the questionnaire for alignment with goals 
and adherence to research-based guidelines for writing questions, and organizing and for- 
matting the questionnaire. It is designed for principals, superintendents, and other state, 
district, and school personnel who need to collect information on attitudes, perceptions, 
or facts to inform their decisions. It is not intended for creating questionnaires to measure 
student cognitive abilities, achievement, or personality traits. 
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Why this guide? 


Education decisionmakers who need information on attitudes, perceptions, or facts in their 
school or community to make good decisions may need to develop original questionnaires 
to survey students, teachers, or parents. But if the questions are not precise, the effort may 
result in inaccurate data and faulty decisions. 

This guide provides a five-step process that educators can follow to develop effective ques- 
tionnaires. It is designed for educators who need to make a decision on an education policy 
or practice but lack the information needed to make that decision and do not already have 
a questionnaire that can be used to gather the information. 


Writing questions may seem easy, but several pitfalls can result in the collection of inac- 
curate information (Burgess, 2001; Thorndike & Thorndike-Christ, 2010). To ensure that 
the time spent creating, administering, and tabulating data from a questionnaire leads 
to accurate and useful information, educators should follow researcbubased principles for 
creating questionnaires. The five-step process in this guide is based on the experience of 
the Wyoming Office of Public Instruction and the Nebraska Department of Education 
working with Regional Educational Laboratory Central on several survey projects (box 1) 
as well as recommendations from experts in the field of questionnaire design and writing 
(Babbie, 1990; Dillman, 2014; Fink, 2013; Phellas, Block, & Seale, 2012; Pope, Boleman, & 
Cummings, 2005). The five steps are: 

1. Determine the goal or goals of the questionnaire. 

2. Define the information needed to address each goal. 


To ensure 
that the time 
spent creating, 
administering, 
and tabulating 
data from a 
questionnaire 
leads to accurate 
and useful 
information, 
educators should 
follow research- 
based principles 
for creating 
questionnaires 


3. Write the questions. 

4. Review the questionnaire for alignment with goals and adherence to research-based 
guidelines for writing questions. 

5. Organize and format the questionnaire. 


Box 1. Basis of the guide 

This guide is based largely on work done in separate projects by Regional Educational Lab- 
oratory (REL) Central in collaboration with the Wyoming Office of Public Instruction and the 
Nebraska Department of Education. Both projects required building knowledge about effective 
questionnaire-writing practices with clients and then creating processes to ensure that those 
practices were incorporated by the clients. 

REL Central assisted the Wyoming Office of Public Instruction in creating questionnaires to 
measure perceptions of education quality in the state. The questionnaires were administered 
to parents, teachers, and community business owners. 

REL Central’s Formative Assessment Research Alliance worked with the Nebraska Depart- 
ment of Education to create questionnaires to measure educators’ use of a statewide forma- 
tive assessment system called Check 4 Learning. With REL Central, Nebraska created three 
questionnaires that were administered to teachers, principals, and district-level assessment 
coordinators. 
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What kind of questionnaires can and cannot be created using this guide? 


Questionnaire is defined here as a set of questions administered to individuals to gather 
useful information. This guide outlines steps to create questionnaires to measure attitudes, 
perceptions, or facts. It is not intended for creating questionnaires to measure achieve- 
ment, cognitive abilities, or personality traits (table 1). 


This guide was created to assist educators in gathering information when making lower 
stakes decisions. For example, the guide is appropriate for developing a questionnaire that 
gathers information from teachers on the frequency and type of their instruction practices, 
on their satisfaction with a new software program, or on the qualities that parents want 
in an afterschool program. It can be used to develop questionnaires that are administered 
either online or in hard copy. 

In contrast, measuring achievement, cognitive abilities, and personality traits requires a 
more formal process than this guide offers. When those types of data are needed, educa- 
tion decisionmakers should seek existing data, investigate the availability of existing ques- 
tionnaires, or hire an expert to develop the needed questionnaire or test. 


This guide was 
created to 
assist educators 
in gathering 
information when 
making lower 
stakes decisions 


This guide discusses how to write questions for questionnaires. It does not include infor- 
mation related to administering questionnaires or to the larger picture of gathering and 
analyzing data. Specifically, the guide does not provide information on sample selection, 
questionnaire administration (including gaining any approvals needed from an institution- 
al review board or district research committee), and data collection, analysis, and presenta- 
tion. Resources that provide such information are provided in table 2. 


Step 1. Determine the goal or goals of the questionnaire 


The first step in questionnaire development is to determine the research goals (Burgess, 
2001; Pope et al., 2005). In other words, what do you want to learn from the questionnaire? 
For what purpose will the data gathered be used? Not clearly defining the goal or goals at 
the outset runs the risk of gathering incomplete, misleading, or nonessential data (Fink, 
2013; Pope et al., 2005). It is easy to get caught up in wanting to know too many things, so 
identify only one or a few goals that are essential to making a decision (Fink, 2013). 


Table 1. This guide is designed to support the development of questionnaires that 
measure attitudes, perceptions, and facts 

Use this guide to create 
questionnaires to gather information on 

Do not use this guide to create 
questionnaires to gather information on 

Attitudes (such as teachers' opinions on the 
strengths of a new curriculum used in a school) 

Achievement (such as a math or reading test) or 
student learning (such as an end-of-unit quiz) 

Perceptions (such as perceived barriers to improved 
test scores in a school) 

Cognitive abilities (such as processing abilities) 

Facts (such as the type and duration of training each 
school’s staff has attended within a district or the 
number of hours worked by staff outside of normal 
work hours) 

Personality traits (such as level of aggression or 
sociability) 

Source: Authors' compilation. 
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Table 2. References for sample selection, questionnaire administration, and data 

collection, analysis, and presentation 


1 Topic 

Resources | 


Dillman (2014) 

Questionnaire administration 

Fink (2013) 

Thorndike & Thorndike-Christ (2010) 

Data collection and analysis 

Fink (2013) 

Groves et al. (2009) 

Data presentation 

Fink (2013) 


Dillman (2014) 

Sample selection 

Fink (2013) 

Groves et al. (2009) 

Source: Authors' compilation. 


For example, a school principal may need to decide whether to continue a certain reading 
program or may want to know how much work teachers are doing outside of school. List 
the goals of the questionnaire and revise the list to ensure that the goals are not redundant 
or nonessential. 


Clearly defining 
the information 
needed ensures 
a connection 
between the goals 
and the questions 


Step 2. Define the information needed to address each goal 


The second step is to look at each goal and generate a list of information needed to address 
that goal (table 3). For example, if the goal is to understand staff perceptions of school 
discipline, step 2 could involve identifying information on the perceived effectiveness of 
current school policies, the consistency of rule enforcement, and so forth. 

As part of this step, decide whether a questionnaire is actually the best method for gath- 
ering each piece of information. Are the data available elsewhere? For example, if an edu- 
cator wants to know how many days of work teachers miss because of illness, it may be 
best to consult work records. However, if a principal wants to understand the perceived 
strengths of a new reading curriculum, a questionnaire would likely be needed. 

Next, review the list to ensure that the items are clear, essential, not redundant, and capture 
all the information needed to address the goal. Clearly defining the information needed 
ensures a connection between the goals and the questions (Phellas et al, 2012). Develop- 
ing a list, as in table 3, will lay out the entire scope and content of the questionnaire. 

Step 3. Write the questions 


Writing the questions involves three substeps: estimating how many questions to include, 
drafting the questions, and conducting an internal review. 

Estimate how many questions to include 

In deciding the number of questions (Burgess, 2001), consider the audience, their interest 
in the topic, and the amount of time required to complete the questionnaire. It may be 
necessary to cut some of the goals during this step. A questionnaire that is too long may 
fatigue respondents and decrease the accuracy of the data. However, if respondents are 
highly motivated to complete the questionnaire (for example, if they have a high interest 
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Table 3. Examples of questionnaire goals and corresponding information needed 


1 Step 1: questionnaire goal 

Step 2: information is needed on: 

Understand staff perceptions of school 
discipline 

Perceived effectiveness of current school policies and procedures 

• Consistency of rule enforcement 

Perceived effectiveness of consequences for misbehavior 

• Communication among staff regarding misbehavior 

Design an afterschool program 

• Activities identified by stakeholders as important 

• Availability of staff after school 

• Available locations 

• Transportation options for students 

Identify perceived strengths of a new 
reading program 

Ease of use 

• Content of the reading passages 

• Quality of illustrations 
Utility of the teacher's manual 
Extent to which the program is scripted 

Determine the amount of work done by 
staff outside of work hours 

Time that each staff member typically leaves school at the end of the day 
Time staff members spend on work-related matters outside of school 

Identify how teachers use a software 
program to create assessments 

Types of assessments that teachers create 

Frequency of assessments that teachers administer using the software 
Types of questions created with the software 

Determine which schools use a schoolwide 
discipline model defined by the district 

Which components of the model are in place 
How long each component has been in place 

• Frequency and types of professional development that the staff has received on the 
model 

• Benefits the school has achieved after using the model 

• Struggles or barriers the school has encountered using the model 

Understand parents’ perceptions of the 
benefits of a parent-teacher conference 

Whether the conference helps parents understand their child’s academic 
performance, behavior performance, and social and emotional skills 
Whether the conference helps parents learn about possible resources and 
extracurricular activities for their child 

Whether the conference provides parents with extended time to meet with the 
teacher and an opportunity to see the classroom and learning environment 


Source: Authors' compilation. 


in its content or in how the data will be used), a longer questionnaire can be created. 
Incentives can encourage respondents to complete a relatively long questionnaire (Fink, 
2013). 

Completing a questionnaire is a cognitive burden for respondents because they are required 
to read the questions, think intently, and respond (Burgess, 2001; Fink, 2013). The higher 
the cognitive burden, the less likely respondents are to answer accurately or to complete 
the questionnaire. The easier the questionnaire, the more useful and accurate the results 
will be. To create an efficient questionnaire with a lower cognitive burden, consider a few 
guidelines: 

• Avoid demanding or time-consuming questions, such as asking respondents to 
rank order 15 items or to read extensive text (Burgess, 2001). 

• Define terms to reduce misinterpretation and the need for respondents to develop 
their own definitions (Dillman, 2014; Thorndike & Thorndike-Christ, 2010). 

• Group questions on similar topics so respondents do not have to jump mentally 
among topics (Dillman, 2014). 

• Provide checkboxes for responses instead of asking respondents to type or write 
responses. For example, ask respondents to check answers (for example: male, 
female) instead of writing the answer. 
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• Place instructions where they are needed, not just at the beginning of the ques- 
tionnaire, so that respondents do not have to recall the directions (Dillman, 2014). 

• For online questionnaires, avoid questions that require scrolling or switching 
screens (Dillman, 2014). 

Draft the questions 

In the first draft, focus primarily on writing questions that will elicit the needed informs- 
tion that was identified in step 2. More than one question may be needed to fully collect 
the information. For example, only one question is needed to find the number of hours per 
day a respondent typically works, but several questions are likely necessary to understand 
the strengths of a reading program. Guidelines to ensure clarity and avoid bias in question- 
naires are provided in box 2. 

Once the initial questions are created to cover the information needed, review the ques- 
tions and select a format for each one — open-ended or closed-ended — taking into account 


Box 2. Guidelines to ensure clarity and avoid bias in questionnaires 

Questionnaire experts have created guidelines for how best to write and format questions (for 
example, see Check & Schutt, 2012; Thorndike & Thorndike-Christ, 2010). They agree that 
questions should be written clearly and have only one intended meaning to avoid confusing 
or misleading respondents (Check & Schutt, 2012; Dillman, 2014; Fink, 2013). Respondents 
should be encouraged to answer questions honestly and should not feel pressured to respond 
in a particular way (Pope et al., 2005). Careful consideration of the words used in the ques- 
tions and the response choices can minimize bias. 

Use simple language 

In wording questions and responses, use simple language and avoid jargon. For example, 
use short words (“Use brave rather than courageous, job concerns rather than work-related 
employment issue s”; Check & Schutt, 2012, p. 166). Spell out abbreviations and acronyms. 
Using shorter words and spelling out abbreviations reduce the cognitive burden on respon- 
dents because they will not have to read long questions, navigate complex words, or recall the 
meaning of an abbreviation (Check & Schutt, 2012; Fink, 2013). 

Avoid double-barreled questions 

Double-barreled questions ask two questions in one. For example, “Do you support spending 
on professional development and technology?” is double barreled because it asks about two 
separate things: professional development and technology. The question should be broken 
into two questions: Do you support spending on professional development? Do you support 
spending on technology? 

Avoid loaded questions 

A loaded question is based on an assumption about the respondent that may or may not be 
true. For example, the question “Flow often do you take supplies from work?” assumes that 
respondents take supplies from work, when in fact they may not. If a respondent who is asked 
“Did you leave work early today?” answers “no,” it may suggest that he or she does leave work 
early, just not today. 

(continued) 
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Box 2. Guidelines to ensure clarity and avoid bias in questionnaires (continued) 

Avoid questions that elicit socially desirable responses 

Researchers may need to ask about sensitive topics, such as diet, exercise habits, grading 
practices, or teaching philosophies. However, the wording of a question may lead respondents 
to answer with socially desirable, yet untrue, responses. For example, asking “Have you ever 
bullied someone?” is a direct question that respondents may not answer honestly to avoid 
being seen unfavorably. Instead, the question can be softened by replacing inflammatory 
words with less harsh language; for example, "Have you ever mistreated someone?” Questions 
about illegal behavior (for example, “Have you ever used a fake I.D.?” or “Have you ever been 
arrested?”), health habits (for example, “How often do you exercise?’’), or biases (for example, 
“Do you believe any stereotypes?” or “Have you ever told a derogatory joke?”) also may not be 
answered honestly. In such cases questions can be modified to ask about behavior instead of 
attitudes. For example, instead of asking “Do you appreciate diversity?’’, which may provoke 
many respondents to say “yes,” ask more specific questions that inquire how often respon- 
dents interact with members of a different background or how often they attend programming 
on diversity-related issues. 

Another way to reduce socially desirable but untrue responses is to ask respondents to use 
a rating scale to rate the extent to which they agree with something. For example, asking “Are you 
a good educator?” and allowing only a yes or no response will lead most people to answer yes 
(Thorndike & Thorndike-Christ, 2010). However, a question that reads “To what extent do you feel 
you are an effective educator?” and provides a rating scale (for example, 1 = strongly effective, 
2 = somewhat effective, 3 = sometimes not effective, 4 = not effective) may produce more honest 
and accurate results (Check & Schutt, 2012; Fink, 2013; Thorndike & Thorndike-Christ, 2010). 

It is usually preferable to use a range when asking about sensitive topics. For example, in 
a question about income, using a range of responses, such as less than $20,000, $20,001- 
$40,000, greater than $40,000, rather than asking for the respondent’s actual income can 
increase the likelihood of an honest answer (Burgess, 2001; Fink, 2013). 

Avoid leading questions 

Avoid questions or response choices that lead people to respond in ways that may not accu- 
rately represent their views. Leading questions indicate that respondents should respond in 
a certain manner by weighting response categories toward one direction or using emotionally 
charged words (Burgess, 2001). For example, a question that reads “Don’t you think teachers 
should be paid more?” and allows only a yes or no response choice suggests answering yes 
is expected. A better format is to ask respondents to rate their agreement with the state- 
ment “Teachers should be paid more” and provide a rating scale (such as: 1 = strongly agree, 
2 = agree, 3 = neither agree nor disagree, 4 = disagree, 5 = strongly disagree). Response 
choices can influence responses, as well. A question that reads “How would you rate the 
instruction in this class?” and provides answer options of “excellent, good, or satisfactory” is 
biased because no negative response options are offered. Finally, words that are emotionally 
charged, political, or exclude a certain culture or population can bias responses. For example, 
words such as lazy, freedom, or equality can influence respondents’ choices and should be 
avoided (Burgess, 2001; Check & Schutt, 2012; Thorndike & Thorndike-Christ, 2010). 

Other considerations 

• Include directions for each type of question. 

• Use positively worded questions; use negative language sparingly. 

(continued) 
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Box 2. Guidelines to ensure clarity and avoid bias in questionnaires (continued) 


• Clearly define all terms that could be misunderstood. 

• Include an “Other” response when the response options may not include all possible 
responses. 

• Provide anchor labels for each anchor, when applicable (an anchor is the possible 
response, such as "1” on a scale of 1 to 5; an anchor label is the value of that anchor, 
such as “2 = strongly disagree”). 

• Ensure rating scale questions have 4-7 anchor points. 

• Ensure response choices are mutually exclusive and exhaustive, when applicable (for 
example, when asking for age, use “20-29, 30-39, ...” instead of “20-30, 30-40, ...”). 

• Eliminate or revise questions that may be culturally insensitive or offensive. 

Source: Burgess, 2001; Check & Schutt, 2012; Dillman, 2014; Fink, 2013; Phellas et al., 2012; Pope et al., 

2005; Thorndike & Thorndike-Christ, 2010 


such factors as time, resources, and whether the information to be gathered is exploratory 
or whether responses of interest are known in advance (table 4). As questions are revised, 
their format may change. Some developers consider the question formats and wording as 
they write the questions, while others find it easier to focus first on the content of the ques- 
tions, then the format, and then the wording. 

Open-ended questions. Open-ended questions ask participants to write their own 
responses (Thorndike & Thorndike-Christ, 2010). For example, respondents write what- 
ever response they choose to a question such as “What do you believe is a strength of the 
new reading program?” 

Because open-ended questions allow respondents to use their own words, they are highly 
flexible (Fink, 2013). They are particularly useful for exploring attitudes, such as the staff’s 
views on a new reading program or on a certain instruction practice. Open-ended ques- 
tions can be used to gather information without limiting responses and thus missing out 
on pertinent information. 


Open-ended 
questions can be 
used to gather 
information 
without limiting 
responses and 
thus missing 
out on pertinent 
information. 
Closed-ended 
questions are 
useful when the 
needed information 
is not exploratory 
and the developer 
is interested in 
certain responses 


However, open-ended questions require more time for coding and analyzing the data than 
do closed-ended questions (Dillman, 2014; Fink, 2013). The analysis time can be consider- 
able, particularly when responses are long. One method to shorten this time is to provide a 
word or character limit for responses, which is enforceable if the questionnaire is adminis- 
tered online (Fink, 2013). 


Open-ended questions can be structured or unstructured (Fink, 2013; Kline, 2005). Struc- 
tured open-ended questions ask for specific information (for example, describe one strength 
of a reading program). Unstructured open-ended questions encourage more in-depth 
responses (for example, explain how you implemented the reading program). 

Closed-ended questions. Closed-ended questions provide limited response options (see 
table 4; Thorndike & Thorndike-Christ, 2010). They are useful when the needed informa- 
tion is not exploratory and the developer is interested in certain responses. For example, 
the open-ended question, “What do you believe are the strengths of the new reading 
program?” allows staff to offer a range of views and attitudes. A closed-ended question 
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Table 4. Questionnaire developers can use a variety of response formats for open-ended and closed- 
ended questions 


Question type 

Considerations for its use 

Example 

Open-ended 


- When gathering information on exploratory topics 
(for example, a principal wants to determine why 
students are not attending afterschool programs 
and does not have a predetermined hypothesis). 
When the range of anticipated responses is 
unknown. 

When more elaborated responses are needed. 

■ Takes longer for data analysis than closed-ended 
questions. 

What do you feel are some of the strengths of the 
new reading intervention program? Use the space 
below to write your response. 

■ What is your favorite thing about teaching in this 
school and why? Use the space below to write your 
response. 

Closed-ended 

Single response 

• When time and resources are limited and 
efficiency is important. 

Helpful for mutually exclusive categories, such as 
grade level or gender. 

Helpful for identifying a primary choice. 

May limit the information gathered if respondents 
want to provide more than one answer. 

• Which of the following is the primary strength of 
the new reading program? Please mark only one. 
The content of the stories 
Integration of writing and spelling 
■ Illustrations in stories 
The number of reading components that the 
program covers 
1 Layout of teacher's manual 

Checklist 

When time and resources are limited and 
efficiency is important. 

When collecting multiple responses to a question. 
• When categories are not mutually exclusive. 

Does not provide information on differences or 
preferences among response choices. 

■ Which of the following are strengths of the new 
reading program? Please mark all that apply. 

• The content of the stories 
Integration of writing and spelling 

• Illustrations in stories 

• The number of reading components that the 
program covers 

• Layout of teacher's manual 

Ranking 

■ When time and resources are limited and 
efficiency is important. 

• When determining the relative importance of 
responses. 

• Requires more time to complete. 

Higher cognitive demand may contribute to 
burnout or fatigue. 

Requires that respondents are knowledgeable in 
the topic and capable of assigning rankings. 

■ Please rate the top 3 strengths of the new reading 
program. Place a “1” by your top choice, a “2” by 
your second choice, and a “3” by your third choice. 

The content of the stories 

Illustrations in stories 

Layout of teacher's manual 

Rating scale 

When time and resources are limited and 
efficiency is important. 

A format familiar to many respondents. 
Preferred for nondichotomous data (data with 

On a scale from 1 to 5, please rate the extent to 
which each of the following features of the new 
reading program is a strength. Circle the number for 
your response. 


more than two values, such as agree, neutral, or 
disagree). 

Requires careful consideration of wording, anchor 
points, and how to organize the data generated 
from responses. 


The content of the stories: 
Not a strength 

12 3 


Strength 

5 


Source: Authors’ compilation. 


could be used if only certain, defined aspects of the reading program are considered 
strengths (such as “Do you believe the illustrations in the story are a strength?”, “Do you 
believe the content of the stories is a strength?”, or “Which of the following strengths do 
you find in the reading program?”). 
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Closed-ended questions are popular because the data generated can be summarized and 
analyzed more efficiently than the data from open-ended questions (Fink, 2013; Kline, 
2005). Closed-ended questions are often used when time and resources for analysis are 
limited (Kline, 2005). However, they may limit the richness and depth of responses and 
the possible answers respondents can offer (Dillman, 2014; Fink, 2013; Thorndike & 
Thorndike-Christ, 2010). To address this issue, an “other” response option may be provid- 
ed so respondents can add items that do not appear among the options (Burgess, 2001). 


In writing closed-ended questions, include the possible range of responses for that question 
plus one or more standard response options that make sense in the context of the question. 
Standard response options include “don’t know” (which allows respondents to indicate that 
they are not sure how to answer and avoids forcing them into a response; Pope et al., 2005), 
“not applicable” (which allows respondents to indicate that the question is not relevant to 
them; Burgess, 2001; Fink, 2013), and “none of the above” or “none” (which allows respon- 
dents to indicate that none of the options is preferable; Fink, 2013; Pope et al., 2005). For 
example, respondents asked to indicate a strength of a reading program may choose “don’t 
know” if they are unsure of its strengths, “not applicable” if they are not familiar with the 
program, or “none” if they do not agree with any of the strengths listed as options. 

Four main formats are used for closed-ended questions: single response, checklist, ranked 
response, and rating scale (Burgess, 2001; Fink, 2013; Thorndike & Thorndike-Christ, 
2010; Whitcomb & Merrell, 2012). 

Single-response. In this format the question provides a list of responses from which respon- 
dents select only one answer. For example, it can be used to ascertain what respondents 
identify as the primary strength of a reading program (see table 3). It is useful for mutually 
exclusive categories (such as “What is your job title in the school?”) or when there are only 
a few response options (generally three to five). However, limiting the response options to 
one may reduce accuracy if respondents wish to mark more than one option. For example, 
if respondents view several facets of a reading program as strengths, forcing them to choose 
only one may misrepresent their view that there are multiple strengths (Fink, 2013; Thorn- 
dike & Thorndike-Christ, 2010). 


The data generated 
from closed-ended 
questions can be 
summarized and 
analyzed more 
efficiently than the 
data from open- 
ended questions, 
but they may 
limit the richness 
and depth of 
responses and the 
possible answers 
respondents 
can offer 


Checklist. In this format the question provides a list of responses from which respondents 
can select multiple answers (Whitcomb & Merrell, 2012). Such questions generally should 
include directions to mark all that apply. In the previous example respondents could iden- 
tify multiple strengths of the reading program by choosing from a list. 

Ranked responses. In this format the question asks respondents to rank the provid- 
ed options, which is beneficial when assessing the relative importance or preference of 
responses (Fink, 2013). For example, a school principal, who has identified three options to 
save funds might offer staff members a ranked-response question to indicate their relative 
preference for those options. 

Rating scale. In this format the question uses a rating scale to measure agreement (for 
example, strongly disagree to strongly agree), importance (for example, not at all import- 
ant to extremely important), frequency (for example, never to always), or satisfaction (for 
example, very dissatisfied to very satisfied). This format is useful when responses are not 
dichotomous (that is, either-or responses; Fink, 2013; Whitcomb & Merrell, 2012). 
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A Likert scale is a specific type of rating scale that is used to measure attitudes and 
uses symmetrical responses. For example, the responses “1 = strongly agree, 2 = agree, 
3 = neither agree nor disagree, 4 = disagree, 5 = strongly disagree” are symmetrical because 
they measure agreement and disagreement in the same manner with using “strongly” on 
both ends. In other words the response options are balanced and have an equal number of 
positive and negative responses. A nonsymmetrical response may be “1 = never, 2 = often, 
3 = usually, and 4 = always” because responses 1 and 2 are not symmetrical options to 
responses 3 and 4 (Dillman, 2014; Fink, 2013). 

A common question about rating scales is how many response options to provide (Fink, 
2013). Four to seven response options are the most common because anything above seven 
choices can create a cognitive burden on the respondents and lead to less reliable results 
(Fink, 2013; Kaplan & Saccuzzo, 2013; Thorndike & Thorndike-Christ, 2010). The choice 
between an odd number and even number of responses depends on the goal of the ques- 
tionnaire and is most relevant for Likert scale questions. An even number of choices will 
force respondents to select one side or the other (a type of forced-choice question; Thorn- 
dike & Thorndike-Christ, 2010). An odd number of response choices will allow respon- 
dents to select a neutral position (Fink, 2013; Thorndike & Thorndike-Christ, 2010). For 
example, if a question asking respondents to rate their agreement with using a reading 
program provides an odd number of choices (for example, 1 = strongly disagree, 2 = dis- 
agree, 3 = neither disagree nor agree, 4 = agree, 5 = strongly agree), respondents can 
choose to be neutral by selecting 3. However, providing an even number of choices (for 
example, 1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree) forces respon- 
dents to either agree or disagree. 

Including a neutral answer or “don’t know” option increases the risk of respondents 
marking those answers because it requires less thinking. However, if there is not a “don’t 
know” option, respondents who cannot or do not want to answer a question may give 
a random answer or stop completing the questionnaire (Fink, 2013; Schwarz & Bohner, 
2001). If some unusable data or incomplete questionnaires can be tolerated, a “don’t know” 
option can be used. If not (for example, because every respondent’s input is necessary 
for making a decision from the data), eliminating “don’t know” options or using an even 
number of rating scale responses is preferred (Krosnick et al., 2002). 

“Don’t know” options are generally recommended for factual questions but not for attitudi- 
nal ones (Krosnick et al., 2002). 

Conduct an internal review 

Once a set of questions and their formats are determined, review the questions to ensure 
that each has only one interpretation and adheres to research-based guidelines for writing 
questions (see box 2). Because questionnaire writers may be too close to the subject matter 
and thus miss items that could confuse respondents, an outside review is desirable, as 
described in step 4- 
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Step 4. Review and revise the questionnaire for alignment with goals 
and adherence to research-based guidelines for writing questions 


The draft questionnaire should be reviewed by people who are not involved in its creation 
to ensure that the questions align with the goals and information needed, follow research- 
based guidelines, and can be easily interpreted by respondents. 

Review the questionnaire 

Identify at least two people to review the questionnaire and provide feedback. Having more 
reviewers provides additional perspectives but can be difficult to manage. Questionnaire 
reviewers can be professionals with content expertise or people similar to the intended 
questionnaire respondents (for example, a teacher could review a questionnaire intended 
for teachers). 

Provide reviewers with a list of the goals and information needed for each goal and ask 
them for feedback on how well each question addresses the goals and information needed 
(see worksheet 1 at the end of this guide). Analyzing the questions and their alignment 
to the goals and information needed helps determine whether the questionnaire gathers 
the desired data (Kaplan & Saccuzzo, 2013) and whether questions follow research-based 
guidelines (see worksheet 2 at the end of this guide). Checking the questions against the 
guidelines helps ensure that they measure what they are intended to measure (Fink, 2013). 

Also ask reviewers to note any other problems they identify in the questionnaire. 

Revise the questionnaire based on reviewer comments 

Read the review comments and make any necessary adjustments to the draft question- 
naire. Reviewers may find that some questions collect unnecessary information or do not 
align with the goals of the study. Flag and revise any questions reviewers indicate did not 
follow research-based guidelines. If necessary, discuss the reviewers’ comments with them 
to better understand their thinking. 

Step 5. Organize and format the questionnaire 


By step 5 the content of the questionnaire is ready and the focus shifts to organizing and 
formatting it to match the administration method (electronic or paper and pencil). 

Determine the method of administration 

Determine whether the questionnaire will be administered electronically or by paper and 
pencil (and if by paper and pencil, whether it will be mailed or administered in person). 
Both options have benefits and drawbacks. Online administration can be more convenient, 
but respondents need access to the necessary technology (Fink, 2013). Paper-and-pencil 
administration can be more expensive because of printing costs and the time involved in 
manual or scanned data entry but does not require much technology. Mailing question- 
naires requires postage and up-to-date addresses, whereas administering the questionnaire 
in person requires supervising personnel (Burgess, 2001; Fink, 2013). 
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Verify rules for consent and permission 


Check with school or district administrators for local policies related to informed consent, 
data collection, and maintenance of confidential information. The Protection of Human 
Subjects in Research Coordinator in the U.S. Department of Education’s Office of the 
Chief Financial Officer offers a helpful website (http://www2.ed.gov/about/offices/list/ 
ocfo/humansub.html). Questionnaires should tell respondents how the data collected will 
or can be used and should include assurances that identifying information will not be 
released publicly. Questionnaires may be coded with a unique number for each respondent 
that is matched with the respondent’s name in a separate place or file that is locked or 
password protected (Fink, 2013). Familiarize yourself with what data are permissible and 
legal to gather in certain settings. For example, sensitive information such as income or 
health history is permissible to ask teachers directly, but district-level personnel likely need 
a teacher’s signed consent before they can access information on income or health history. 

Format the questionnaire 

The questionnaire format should be easy to follow and visually appealing. Considerations 
for organizing and formatting a questionnaire include (Fanning, 2005): 

• Cover sheet. A cover sheet should clearly state the purpose of the questionnaire 
as well as the name and contact information of the administering organization in 
case respondents have questions. 

• Informed consent. A paragraph or separate page should explain how the data will 
be used. Depending on local requirements, it can: 

o Ask for permission to collect data or state that completing the questionnaire 
grants permission for the data to be used, 
o Explain that participation is voluntary (when applicable). 

° State that the respondent can stop the questionnaire at any time without 
penalty or judgment. 

o Explain how the respondent’s answers will be kept confidential and that idem 
tifying information will not be released publicly. 

• Directions. Directions should be stated clearly for each group of questions. Respom 
dents should not be burdened with following complex directions (Dillman, 2014). 

• Back sheet. On the final page, a space for additional comments can be added. 

• Question order. The order of questions should ensure that respondents are moti- 
vated to complete the questionnaire. To consider: 

o Place neutral, “friendlier” questions (for example, demographic questions) at 
the beginning to build rapport with the respondent (Dillman, 2014). 

° Place sensitive questions (about salary or employment, for example) in the middle. 
° Place shorter questions at the end where respondents’ motivation and energy 
may be flagging (Burgess, 2001; Pope et al., 2005). 

° When possible, provide markers, such as a percentage of the total number of 
questions completed for each page (when electronic) or a note at the end of a 
page that says “you’re halfway done!”, to indicate progress through the ques- 
tionnaire (Dillman, 2014; Fink, 2013). 

• Layout. Font size, indentation, and appearance should be consistent (Fanning, 
2005; Fink, 2013). To consider: 

o Organize questions so the questionnaire appears brief, easy-to-complete, and 
easy-to-read on paper or on a computer screen. 
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o Provide more white space between questions than within the response options 
of those questions (Burgess, 2001). 

o Use contrasting colors to delineate different sections of the questionnaire 
(such as black on white: dark type for questions and lighter type for responses; 
Dillman, 2014). 

° Avoid landscape orientation (as opposed to portrait orientation) because it is 
more difficult to read. 

o Lay out the questionnaire with a symmetrical look and about the same number 
of questions per page (Phellas et al, 2012). 
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Worksheet 1. Do the questions address the goals and information needed? 


Examine each question and determine whether each question addresses any of the questionnaire’s goals or 
information needed. 


Mark in the worksheet below any questions that do not appear to align with the goals of the questionnaire or 
information needed and any comments on why. 


Question 

number 

Goal 

Comments 
























































Additional comments or concerns with questions: 
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Worksheet 2. Do the questions follow research-based guidelines? 


Review each question against the guidelines below. Indicate questions that do not follow a guideline and 
provide any relevant comments. 


Guideline 

Question 
numbers that 
do not follow 
the guideline 

Comments 

Question wording 



Uses simple, jargon-free vocabulary 



Uses standard, nontechnical language 



Includes appropriate directions for the question type 



Asks only one question (as opposed to a double-barreled 
question that asks two questions in one) 



Uses positive wording, with negatives used sparingly 



Clearly defines all terms that could be misunderstood 



Question responses 



Provides an “Other” response, if applicable 



Provides labels for each anchor (an anchor is the possible 
response, such as “1” on a scale of 1 to 5; an anchor label is 
the value of that anchor, such as “2 = strongly disagree") 



Provides four to seven response options (when asking for a 
rating) 



Provides mutually exclusive response choices 



Provides exhaustive response choices 



Minimizing bias 



Avoids culturally insensitive or offensive language 



Avoids biased or emotionally charged words 



Avoids assumptions about the respondent (loaded question) 



Avoids soliciting a socially desirable answer 



Provides ranges for response choices (when question is 
potentially sensitive) 
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